This invention relates to the art of creating and of processing images. It provides a method and apparatus for computing or processing visually perceptible images in terms of a lightness field. A lightness field is herein defined as the output of a process that uses radiances falling on a light-detector from an original image to produce a new set of values that correspond to the sensations of lightness produced by the human visual system.
Vision science begins with the basic properties of light, for which there are clearly established quantitative concepts tied to physical measures. For example, the radiance from a given region denotes flux of radiant energy per unit projected area per unit solid angle. Reflectance of a surface denotes the fraction of incident radiant energy (of a specified wavelength distribution) reflected by the surface.
Less consensus has developed in characterizing human reaction to light, and it is not surprising that a term such as lightness, which generally refers to a sensation, has at different times been assigned different meanings. Thus, in Webster's New International Dictionary, Second Edition, lightness was defined as the "state or quality of illumination, or degree of illumination," i.e., a physical measure. In Webster's New International Dictionary, Third Edition, lightness in addition denotes a sensation, namely, "the attribute of object colors by which the object appears to reflect or transmit more or less of the incident light and which varies for surface colors from black as a minimum to white as a maximum. . .".
The later definition recognizes the importance of appearance, i.e. the sensation. Sensation is important because the lightness of an object is not necessarily related to the physical quantity of light from the object, either in radiometric or photometric terms. An object will hold its position on a lightness scale despite large changes in its intensity. Much of the difficulty in terminology thus arises because visual sensations characterizing a specific region cannot be directly related to any physical measure of light from that region alone.
The term lightness as used in connection with these teachings will have a primary meaning of a visual sensation as produced by biological systems such as human vision. These lightness sensations are produced by a biological system that takes the radiance at each point in the field of view and yields a lightness value for each point in the field of view. In particular, lightness denotes a visual sensation which ranges from dark to light and which characterizes image regions in specific conditions. One of the more interesting properties of human vision is that the cerebro-retinal lightness signal processing system is such that the lightness sensed at any point does not have a simple functional relationship to the radiance at that point. Lightness thus does not depend on the physical properties of single points or objects in the field; lightness instead depends on relationships between physical properties of all the points or objects across the field of view. Lightness does not result from point by-point processing; lightness results from processing the entire field.
Lightness can be quantified by employing a technique of visual comparisons. First, one establishes a standard display that includes reference areas covering the range from minimum to maximum reflectance in controlled illumination and surround. One then presents an observer with another area in any viewing condition and asks the observer to select the best visual match of that area to a reference area of the standard display. Finally, one takes the reflectance of the chosen reference area and typically applies a monotonic scaling function so that equal increments in the resulting lightness numbers are assigned to equal changes in sensation. Such approach emphasizes the fact that although lightness is a sensation produced by a human or other biological system, it is a quantifiable entity. The correspondence in reports from large numbers of observers in numerous experiments of this type shows that these sensations are generated by a repeatable set of physical relationships. Since lightness depends on the entire image, a physical definition of lightness must incorporate a process which utilizes the entire field of view.
The teaching herein describes signal processing systems which generate quantities that correspond to lightness. The quantities, however, are generated by machine signal processing systems rather than biological systems. For clarity we define a separate term to described these machine-generated quantities that correspond to lightness. We have chosen the term "lightness field" as the name of the output of the machine for the selected field of view. The choice emphasizes the fact that a lightness field is derived from signal processing operations which involve the field of view. This characteristic of lightness field computation distinguishes it from other signal processing strategies that involve either single points or local areas of the image.
Human vision is remarkable for its ability to generate sensations that correspond to the physical properties of objects in the field of view regardless of the radiant intensity and of the wavelength distribution of the light falling on the retina. The wavelength-intensity distribution of the light from an object falling on a light detector such as a photosensitive element is a function of two independent variables: the illumination at the object and the ability of the object to reflect or transmit light. However, the radiance measurements for any single picture element, i.e. pixel, are not subject to an analysis which identifies the independent contributions of illumination and of object properties.
This invention, on the other hand, uses the entire field of view to calculate visual properties of objects substantially independently of the properties of the illuminant. Using the entire field of view is considered essential to a solution of the problem that cannot presently be solved by processing information at individual pixels independently of that at other pixels.
It is difficult for a photograph or like image to accommodate variabilities of lighting conditions, even when care is taken to center the limited dynamic range of the image medium on the dynamic range of the light being recorded. Consider the light reflected from a collection of different colored and textured objects, ranging from the brightest white to the darkest black, when special effort is taken to illuminate the collection so that the same intensity of light of the same spectral composition falls on each point in the field of view. The dynamic range of the light reflected from this collection of uniformly-illuminated objects is significantly less than a range of 100-to-1. The brightest white objects may reflect roughly only 92% of the light falling on them, whereas the darkest black velvet objects may reflect roughly at least 3% of the light falling on them. The light reflected from objects having matte surface falls between these extreme values for bright white and for black velvet.
These physical properties of objects limit reflective reproduction media, such a photographic prints and printing, to a like dynamic range, i.e. to a range significantly less than 100-to-1.
However, the dynamic range of intensities from real life, i.e. from natural images, is far larger than that in this special uniformly-illuminated experiment. Natural scenes include sizable variations in the dynamic range of the illumination. First, natural illumination varies both in overall total intensity and in local regions because some objects are shaded by others. Second, the spectral composition of the incident light may vary dramatically from skylight to sunlight to tungsten light to fluorescent light. As noted, human vision is remarkable in that it generates image sensations which are nearly indifferent to this extreme variability of lighting conditions. These same variations in illumination, however, produce marked and usually detrimental results in conventional image-reproducing systems, whether photographic, television or printing.
The present invention endeavors to resolve these imaging problems. More particularly, this invention provides mechanisms that detect the large dynamic range of light intensities, that use a novel strategy to calculate approximations of visual properties of the objects in the field of view, and that represent the entire image in a limited dynamic range that is optimal for media such as photography, television and printing. A significant feature of the invention accordingly is the calculation of lightness fields that portray large dyanmic ranges of the original scene in terms of limited dynamic ranges defined by the range of intensities available in various media.
Various photographic defects result from attempting to photograph the natural environment "as is". Ordinarily the photographer consciously tries to avoid or minimize these defects by the practice of his art. He measures the light coming from the objects in the scene and adjusts the time and the aperture settings so that the exposure will fall on the desired portion of the limited dynamic range of the film. He artificially illuminates all or part of the scene to compensate for non-uniformities in illumination across the scene. He uses color-correcting filters to match the spectral properties of the scene to the spectral sensitivity of the film. The photographer makes these corrections in part by estimating the physical properties of the illumination, perhaps with the aid of a light meter. A television cameraman and his crew follow similar procedures. Further, present-day automatic cameras determine the lens aperture and the shutter time settings, but they do not do all that is necessary to correct the range of lighting problems found in a natural environment.
The power of the concepts set forth herein can be illustrated by the following practical experiments demonstrating advantages realized and realizable in one practice of this invention. The description is of six experiments that emphasize typical common handicaps presently encountered in photographing complex images. Typical photographic defects result from the mismatch between the dynamic color range of an orginal scene and the limited color and intensity responses of photographic materials. For the following experiments, a complex original scene is provided in the form of a recorded and displayed television image. This image is in full color and portrays a wide range of hues occurring in varying densities, for example, a woman in a colorful costume against a bright multicolored floral background.
In each experiment a control image is described which represents the response to each original scene of a conventional photographic system that does not employ this invention. The first such control image demonstrates the mismatch commonly encountered between the dynamic color range of an original scene and the limited color response characteristic of color film. For example, highlights exhibit a degree of levelling and desaturation, whereas shadow areas show little evident image detail. In the first experiment of the invention, the same original scene is subjected to lightness analysis by the lightness imaging system defined below and is photographed on a standard photographic medium. This first processed image is found to possess much clearer image detail in shadow and in highlight areas, a better defined range of color values, and improved saturation. To the eye of an observer, the processed image more accurately represents the content of the original scene than does the control image. In producing the processed image in this first experiment, as in the others described below, the only image information available to the lightness imaging system is that which is contained within the original scene itself.
In a second experiment, the same original scene used in the first experiment is modified by the superimposition of a ten-to-one illumination gradient from one side of the scene to the other. When this modified scene is photographed, using conventional techniques to produce a control image, most of the image detail is lost in the darkest portions of the image, or in the brightest portions, and most of the color values are lost. But when this modified scene is analyzed and photographed using the lightness imaging system of this invention, a second processed color print is obtained which is virtually indistinguishable from the first processed image described above. The ten-to-one illumination gradient has disappeared, and the resultant image displays the same saturation, image detail, and pleasing dynamic range as that of the first processed image. Furthermore, this second processed image is obtained by the same lightness imaging system operating in the same way and with no further modifications, adjustments, or revised programming.
A third experiment is performed. The original scene is now subjected to different modification representing tungsten illumination of the scene. As a consequence, the intensity of the middle-wavelength illuminant is only 41% of that of the long-wavelength illuminant, and that of the short-wavelength illuminant is a mere 5% of that of the long-wavelength illuminant. An ordinary photograph of this modified scene is strongly reddish with few discernible green color values and with practically no visible blue color values. However, when this modified scene is processed by the lightness imaging system, operating in the same unmodified way, a third processed color print is obtained which is virtually indistinguishable from the first two.
Then in a fourth experiment, the original scene is subjected to both of the illumination modifications employed in the second and third experiments. Thus, not only are the color values of the entire scene altered by a tungsten illuminant, but the illuminant varies by a ten-to-one gradient from one side of the original scene to the other. A conventional photograph of this modified scene is strongly reddish with few discernible green color values and practically no blues, and all the image detail appears lost in the darker portion of the illumination gradient. At this point it should come as no surprise to learn that indeed the fourth processed image obtained by the apparatus and method of this invention is not only essentially free of the imposed modifications, but is substantially identical to the first, second, and third processed images.
A common problem in photography, different from those already considered, is that of preserving image detail in distinct areas of a scene that has different overall levels of illumination. Two additional experiments are described with a new original scene that shows a household interior in which a person is seated by a window onto a colorful outdoor view.
In a fifth experiment, the new original scene is characterized by an eight-fold reduction in the illumination of the view outside the window; that is, this modified original scene depicts the illumination of an evening. When this evening scene is photographed, with the same conventional practices previously used to produce a control image, most of the image detail and color values in the outdoor portion of the scene are lost. But when this modified scene is analyzed and photographed using the lightness imaging system of this invention, a fifth processed color print is obtained in which the scene is accurately represented both inside and outside the window with the same improvements in image quality described for the previous experiments and in which the outdoor view still appears somewhat darker, as is true of the evening setting.
In a sixth experiment, the new original scene is characterized by an eight-fold reduction in the illumination of the indoor scene in front of the window with no reduction in the illumination of the outdoor view behind the window. The modified scene now represents a daytime setting with the indoor portion relatively darker than the bright outdoor view. When this daytime setting is photographed, most of the image detail and color values in the indoor portion are lost. But when this modified scene is analyzed and photographed using the lightness imaging system of this invention, a sixth processed color print is obtained in which the scene is accurately represented both inside and outside the window with the same improvements in image quality described for the previous experiments and in which the indoor scene appears somewhat darker, as is the actual case for a daytime setting.
Furthermore, the fifth and the sixth processed images are obtained with exactly the same lightness imaging system operating in exactly the same way as for the first four processed images.
The invention thus advances the art of retinex processing as disclosed in the literature, examples of which are:
U.S. Pat. No. 3,553,360 PA1 U.S. Pat. No. 3,651,252 PA1 E. H. Land and J. J. McCann, "Lightness and Retinex Theory", J. Opt. Soc., Am., 61, 1-11 (1971). PA1 E. H. Land, "The Retinex Theory of Colour Vision", Proc. Royal Inst. of Gr. Brit., 47 (1974). PA1 J. J. McCann, S. P. McKee and T. H. Taylor, "Quantitative Studies in Retinex Theory", Vision Research, 16, 445-458 (1976).
Other publications in the imaging art are the article by T. G. Stockham, Jr., "Image Processing in the Context of a Visual Model", Proceedings of the IEEE, Vol. 60, No. 7, July 1972, pages 828 through 842; the article by David Marr, "The Computation of Lightness by the Primate Retina", Vision Research, Vol. 14, pages 1377 through 1388; and the article by Oliver D. Faugeras, "Digital Color Image Processing Within the Framework of a Human Visual Model", IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. ASSP27, No. 4, August 1979, pages 380-393. This invention employs techniques which differ significantly from the image processing which these articles discuss.
Objects of this invention, and advantages which it brings to the art of imaging, include attaining lightness imaging with fewer signal processing steps or computations in considerably less time then previously available.
A further object is to provide a method and apparatus for lightness imaging applicable on a practical basis to numerous image processing and numerous image creating instances.
Another object of the invention is to provide a method and apparatus for providing an image, termed a lightness image, which represents a scene in a limited dynamic range that is optimal for display media such as photography, television and printing.
It is also an object to provide image processing that uses information acquired at one segmental area of an image in evaluating information acquired at other segmental areas in a learning-like manner that attains a desired lightness field in relatively small time and with relatively few processing steps.
It is also an object to provide a method and apparatus of the above character suited for commercial application.
Other objects of the invention will in part be obvious and will in part appear hereinafter.